Wrapper Induction for Information Extraction

Wrapper Induction for Information Extraction by Nicholas Kushmerick Chairperson of Supervisory Committee: Professor Daniel S. Weld Department of Computer Science and Engineering The Internet presents numerous sources of useful information|telephone directories, product catalogs, stock quotes, weather forecasts, etc. Recently, many systems have been built that automatically gather and manipulate such information on a user's behalf. However, these resources are usually formatted for use by people (e.g., the relevant content is embedded in HTML pages), so extracting their content is di cult. Wrappers are often used for this purpose. A wrapper is a procedure for extracting a particular resource's content. Unfortunately, hand-coding wrappers is tedious. We introduce wrapper induction, a technique for automatically constructing wrappers. Our techniques can be described in terms of three main contributions. First, we pose the problem of wrapper construction as one of inductive learning . Our algorithm learns a resource's wrapper by reasoning about a sample of the resource's pages. In our formulation of the learning problem, instances correspond to the resource's pages, a page's label corresponds to its relevant content, and hypotheses correspond to wrappers. Second, we identify several classes of wrappers which are reasonably useful, yet e ciently learnable. To assess usefulness, we measured the fraction of Internet resources that can be handled by our techniques. We nd that our system can learn wrappers for 70% of the surveyed sites. Learnability is assessed by the asymptotic complexity of our system's running time; most of our wrapper classes can be learned in time that grows as a small-degree polynomial. Third, we describe noise-tolerant techniques for automatically labeling the examples. Our system takes as input a library of recognizers, domain-speci c heuristics for identifying a page's content. We have developed an algorithm for automatically corroborating the recognizer's evidence. Our algorithm perform well, even when the recognizers exhibit high levels of noise. Our learning algorithm has been fully implemented. We have evaluated our system both analytically (with the PAC learning model) and empirically. Our system requires 2 to 44 examples for e ective learning, and takes about ten seconds of CPU time for most sites. We conclude that wrapper induction is a feasible solution to the scaling problems inherent in the use of wrappers by information-integration systems.